<!-- HTML header for doxygen 1.8.6-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<title>OpenCV: Use OpenCL in Android camera preview based CV application</title>
<link href="../../opencv.ico" rel="shortcut icon" type="image/x-icon" />
<link href="../../tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../jquery.js"></script>
<script type="text/javascript" src="../../dynsections.js"></script>
<script type="text/javascript" src="../../tutorial-utils.js"></script>
<link href="../../search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../search/searchdata.js"></script>
<script type="text/javascript" src="../../search/search.js"></script>
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
    jax: ["input/TeX","output/HTML-CSS"],
});
//<![CDATA[
MathJax.Hub.Config(
{
  TeX: {
      Macros: {
          matTT: [ "\\[ \\left|\\begin{array}{ccc} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{array}\\right| \\]", 9],
          fork: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ \\end{array} \\right.", 4],
          forkthree: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ \\end{array} \\right.", 6],
          forkfour: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ #7 & \\mbox{#8}\\\\ \\end{array} \\right.", 8],
          vecthree: ["\\begin{bmatrix} #1\\\\ #2\\\\ #3 \\end{bmatrix}", 3],
          vecthreethree: ["\\begin{bmatrix} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{bmatrix}", 9],
          cameramatrix: ["#1 = \\begin{bmatrix} f_x & 0 & c_x\\\\ 0 & f_y & c_y\\\\ 0 & 0 & 1 \\end{bmatrix}", 1],
          distcoeffs: ["(k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6 [, s_1, s_2, s_3, s_4[, \\tau_x, \\tau_y]]]]) \\text{ of 4, 5, 8, 12 or 14 elements}"],
          distcoeffsfisheye: ["(k_1, k_2, k_3, k_4)"],
          hdotsfor: ["\\dots", 1],
          mathbbm: ["\\mathbb{#1}", 1],
          bordermatrix: ["\\matrix{#1}", 1]
      }
  }
}
);
//]]>
</script><script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js"></script>
<link href="../../doxygen.css" rel="stylesheet" type="text/css" />
<link href="../../stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<!--#include virtual="/google-search.html"-->
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectlogo"><img alt="Logo" src="../../opencv-logo-small.png"/></td>
  <td style="padding-left: 0.5em;">
   <div id="projectname">OpenCV
   &#160;<span id="projectnumber">4.5.2</span>
   </div>
   <div id="projectbrief">Open Source Computer Vision</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "../../search",false,'Search');
</script>
<script type="text/javascript" src="../../menudata.js"></script>
<script type="text/javascript" src="../../menu.js"></script>
<script type="text/javascript">
$(function() {
  initMenu('../../',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="../../d9/df8/tutorial_root.html">OpenCV Tutorials</a></li><li class="navelem"><a class="el" href="../../df/d65/tutorial_table_of_content_introduction.html">Introduction to OpenCV</a></li>  </ul>
</div>
</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">Use OpenCL in Android camera preview based CV application </div>  </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p><b>Prev Tutorial:</b> <a class="el" href="../../d5/df8/tutorial_dev_with_OCV_on_Android.html">Android Development with OpenCV</a></p>
<p><b>Next Tutorial:</b> <a class="el" href="../../d0/db2/tutorial_macos_install.html">Installation in MacOS</a></p>
<table class="doxtable">
<tr>
<th align="right"></th><th align="left"></th></tr>
<tr>
<td align="right">Original author </td><td align="left">Andrey Pavlenko </td></tr>
<tr>
<td align="right">Compatibility </td><td align="left">OpenCV &gt;= 3.0 </td></tr>
</table>
<dl class="section warning"><dt>Warning</dt><dd>This tutorial is deprecated.</dd></dl>
<p>This guide was designed to help you in use of <a href="https://www.khronos.org/opencl/">OpenCL &trade;</a> in Android camera preview based CV application. It was written for <a href="http://developer.android.com/tools/help/adt.html">Eclipse-based ADT tools</a> (deprecated by Google now), but it easily can be reproduced with <a href="http://developer.android.com/tools/studio/index.html">Android Studio</a>.</p>
<p>This tutorial assumes you have the following installed and configured:</p>
<ul>
<li>JDK</li>
<li>Android SDK and NDK</li>
<li>Eclipse IDE with ADT and CDT plugins</li>
</ul>
<p>It also assumes that you are familiar with Android Java and JNI programming basics. If you need help with anything of the above, you may refer to our <a class="el" href="../../d9/d3f/tutorial_android_dev_intro.html">Introduction into Android Development</a> guide.</p>
<p>This tutorial also assumes you have an Android operated device with OpenCL enabled.</p>
<p>The related source code is located within OpenCV samples at <a href="https://github.com/opencv/opencv/tree/master/samples/android/tutorial-4-opencl/">opencv/samples/android/tutorial-4-opencl</a> directory.</p>
<h2>Preface </h2>
<p>Using <a href="https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units">GPGPU</a> via OpenCL for applications performance enhancements is quite a modern trend now. Some CV algo-s (e.g. image filtering) run much faster on a GPU than on a CPU. Recently it has become possible on Android OS.</p>
<p>The most popular CV application scenario for an Android operated device is starting camera in preview mode, applying some CV algo to every frame and displaying the preview frames modified by that CV algo.</p>
<p>Let's consider how we can use OpenCL in this scenario. In particular let's try two ways: direct calls to OpenCL API and recently introduced OpenCV T-API (aka <a href="https://docs.google.com/presentation/d/1qoa29N_B-s297-fp0-b3rBirvpzJQp8dCtllLQ4DVCY/present">Transparent API</a>) - implicit OpenCL accelerations of some OpenCV algo-s.</p>
<h2>Application structure </h2>
<p>Starting Android API level 11 (Android 3.0) <a href="http://developer.android.com/reference/android/hardware/Camera.html">Camera API</a> allows use of OpenGL texture as a target for preview frames. Android API level 21 brings a new <a href="http://developer.android.com/reference/android/hardware/camera2/package-summary.html">Camera2 API</a> that provides much more control over the camera settings and usage modes, it allows several targets for preview frames and OpenGL texture in particular.</p>
<p>Having a preview frame in an OpenGL texture is a good deal for using OpenCL because there is an <a href="https://www.khronos.org/registry/cl/sdk/1.2/docs/man/xhtml/cl_khr_gl_sharing.html">OpenGL-OpenCL Interoperability API (cl_khr_gl_sharing)</a>, allowing sharing OpenGL texture data with OpenCL functions without copying (with some restrictions of course).</p>
<p>Let's create a base for our application that just configures Android camera to send preview frames to OpenGL texture and displays these frames on display without any processing.</p>
<p>A minimal <code>Activity</code> class for that purposes looks like following:</p>
<div class="fragment"><div class="line"><span class="keyword">public</span> <span class="keyword">class </span>Tutorial4Activity <span class="keyword">extends</span> Activity {</div><div class="line"></div><div class="line">    <span class="keyword">private</span> MyGLSurfaceView mView;</div><div class="line"></div><div class="line">    @Override</div><div class="line">    <span class="keyword">public</span> <span class="keywordtype">void</span> onCreate(Bundle savedInstanceState) {</div><div class="line">        super.onCreate(savedInstanceState);</div><div class="line">        requestWindowFeature(Window.FEATURE_NO_TITLE);</div><div class="line">        getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN,</div><div class="line">                WindowManager.LayoutParams.FLAG_FULLSCREEN);</div><div class="line">        getWindow().setFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON,</div><div class="line">                WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);</div><div class="line">        setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE);</div><div class="line"></div><div class="line">        mView = <span class="keyword">new</span> MyGLSurfaceView(<span class="keyword">this</span>);</div><div class="line">        setContentView(mView);</div><div class="line">    }</div><div class="line"></div><div class="line">    @Override</div><div class="line">    <span class="keyword">protected</span> <span class="keywordtype">void</span> onPause() {</div><div class="line">        mView.onPause();</div><div class="line">        super.onPause();</div><div class="line">    }</div><div class="line"></div><div class="line">    @Override</div><div class="line">    <span class="keyword">protected</span> <span class="keywordtype">void</span> onResume() {</div><div class="line">        super.onResume();</div><div class="line">        mView.onResume();</div><div class="line">    }</div><div class="line">}</div></div><!-- fragment --><p>And a minimal <code>View</code> class respectively:</p>
<div class="fragment"><div class="line"><span class="keyword">public</span> <span class="keyword">class </span>MyGLSurfaceView <span class="keyword">extends</span> GLSurfaceView {</div><div class="line"></div><div class="line">    MyGLRendererBase mRenderer;</div><div class="line"></div><div class="line">    <span class="keyword">public</span> MyGLSurfaceView(Context context) {</div><div class="line">        super(context);</div><div class="line"></div><div class="line">        <span class="keywordflow">if</span>(android.os.Build.VERSION.SDK_INT &gt;= 21)</div><div class="line">            mRenderer = <span class="keyword">new</span> Camera2Renderer(<span class="keyword">this</span>);</div><div class="line">        <span class="keywordflow">else</span></div><div class="line">            mRenderer = <span class="keyword">new</span> CameraRenderer(<span class="keyword">this</span>);</div><div class="line"></div><div class="line">        setEGLContextClientVersion(2);</div><div class="line">        setRenderer(mRenderer);</div><div class="line">        setRenderMode(GLSurfaceView.RENDERMODE_WHEN_DIRTY);</div><div class="line">    }</div><div class="line"></div><div class="line">    @Override</div><div class="line">    <span class="keyword">public</span> <span class="keywordtype">void</span> surfaceCreated(SurfaceHolder holder) {</div><div class="line">        super.surfaceCreated(holder);</div><div class="line">    }</div><div class="line"></div><div class="line">    @Override</div><div class="line">    <span class="keyword">public</span> <span class="keywordtype">void</span> surfaceDestroyed(SurfaceHolder holder) {</div><div class="line">        super.surfaceDestroyed(holder);</div><div class="line">    }</div><div class="line"></div><div class="line">    @Override</div><div class="line">    <span class="keyword">public</span> <span class="keywordtype">void</span> surfaceChanged(SurfaceHolder holder, <span class="keywordtype">int</span> format, <span class="keywordtype">int</span> w, <span class="keywordtype">int</span> h) {</div><div class="line">        super.surfaceChanged(holder, format, w, h);</div><div class="line">    }</div><div class="line"></div><div class="line">    @Override</div><div class="line">    <span class="keyword">public</span> <span class="keywordtype">void</span> onResume() {</div><div class="line">        super.onResume();</div><div class="line">        mRenderer.onResume();</div><div class="line">    }</div><div class="line"></div><div class="line">    @Override</div><div class="line">    <span class="keyword">public</span> <span class="keywordtype">void</span> onPause() {</div><div class="line">        mRenderer.onPause();</div><div class="line">        super.onPause();</div><div class="line">    }</div><div class="line">}</div></div><!-- fragment --><p><b>Note</b>: we use two renderer classes: one for legacy <a href="http://developer.android.com/reference/android/hardware/Camera.html">Camera</a> API and another for modern <a href="http://developer.android.com/reference/android/hardware/camera2/package-summary.html">Camera2</a>.</p>
<p>A minimal <code>Renderer</code> class can be implemented in Java (OpenGL ES 2.0 <a href="http://developer.android.com/reference/android/opengl/GLES20.html">available</a> in Java), but since we are going to modify the preview texture with OpenCL let's move OpenGL stuff to JNI. Here is a simple Java wrapper for our JNI stuff:</p>
<div class="fragment"><div class="line"><span class="keyword">public</span> <span class="keyword">class </span>NativeGLRenderer {</div><div class="line">    <span class="keyword">static</span></div><div class="line">    {</div><div class="line">        System.loadLibrary(<span class="stringliteral">&quot;opencv_java4&quot;</span>); <span class="comment">// comment this when using OpenCV Manager</span></div><div class="line">        System.loadLibrary(<span class="stringliteral">&quot;JNIrender&quot;</span>);</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="keyword">public</span> <span class="keyword">static</span> native <span class="keywordtype">int</span> initGL();</div><div class="line">    <span class="keyword">public</span> <span class="keyword">static</span> native <span class="keywordtype">void</span> closeGL();</div><div class="line">    <span class="keyword">public</span> <span class="keyword">static</span> native <span class="keywordtype">void</span> drawFrame();</div><div class="line">    <span class="keyword">public</span> <span class="keyword">static</span> native <span class="keywordtype">void</span> changeSize(<span class="keywordtype">int</span> width, <span class="keywordtype">int</span> height);</div><div class="line">}</div></div><!-- fragment --><p>Since <code>Camera</code> and <code>Camera2</code> APIs differ significantly in camera setup and control, let's create a base class for the two corresponding renderers:</p>
<div class="fragment"><div class="line"><span class="keyword">public</span> <span class="keyword">abstract</span> <span class="keyword">class </span>MyGLRendererBase <span class="keyword">implements</span> GLSurfaceView.Renderer, SurfaceTexture.OnFrameAvailableListener {</div><div class="line">    <span class="keyword">protected</span> <span class="keyword">final</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga1f6634802eeadfd7245bc75cf3e216c2">String</a> LOGTAG = <span class="stringliteral">&quot;MyGLRendererBase&quot;</span>;</div><div class="line"></div><div class="line">    <span class="keyword">protected</span> SurfaceTexture mSTex;</div><div class="line">    <span class="keyword">protected</span> MyGLSurfaceView mView;</div><div class="line"></div><div class="line">    <span class="keyword">protected</span> <span class="keywordtype">boolean</span> mGLInit = <span class="keyword">false</span>;</div><div class="line">    <span class="keyword">protected</span> <span class="keywordtype">boolean</span> mTexUpdate = <span class="keyword">false</span>;</div><div class="line"></div><div class="line">    MyGLRendererBase(MyGLSurfaceView view) {</div><div class="line">        mView = view;</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="keyword">protected</span> <span class="keyword">abstract</span> <span class="keywordtype">void</span> openCamera();</div><div class="line">    <span class="keyword">protected</span> <span class="keyword">abstract</span> <span class="keywordtype">void</span> closeCamera();</div><div class="line">    <span class="keyword">protected</span> <span class="keyword">abstract</span> <span class="keywordtype">void</span> setCameraPreviewSize(<span class="keywordtype">int</span> width, <span class="keywordtype">int</span> height);</div><div class="line"></div><div class="line">    <span class="keyword">public</span> <span class="keywordtype">void</span> onResume() {</div><div class="line">        Log.i(LOGTAG, <span class="stringliteral">&quot;onResume&quot;</span>);</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="keyword">public</span> <span class="keywordtype">void</span> onPause() {</div><div class="line">        Log.i(LOGTAG, <span class="stringliteral">&quot;onPause&quot;</span>);</div><div class="line">        mGLInit = <span class="keyword">false</span>;</div><div class="line">        mTexUpdate = <span class="keyword">false</span>;</div><div class="line">        closeCamera();</div><div class="line">        <span class="keywordflow">if</span>(mSTex != null) {</div><div class="line">            mSTex.release();</div><div class="line">            mSTex = null;</div><div class="line">            NativeGLRenderer.closeGL();</div><div class="line">        }</div><div class="line">    }</div><div class="line"></div><div class="line">    @Override</div><div class="line">    <span class="keyword">public</span> <span class="keyword">synchronized</span> <span class="keywordtype">void</span> onFrameAvailable(SurfaceTexture surfaceTexture) {</div><div class="line">        <span class="comment">//Log.i(LOGTAG, &quot;onFrameAvailable&quot;);</span></div><div class="line">        mTexUpdate = <span class="keyword">true</span>;</div><div class="line">        mView.requestRender();</div><div class="line">    }</div><div class="line"></div><div class="line">    @Override</div><div class="line">    <span class="keyword">public</span> <span class="keywordtype">void</span> onDrawFrame(GL10 gl) {</div><div class="line">        <span class="comment">//Log.i(LOGTAG, &quot;onDrawFrame&quot;);</span></div><div class="line">        <span class="keywordflow">if</span> (!mGLInit)</div><div class="line">            <span class="keywordflow">return</span>;</div><div class="line"></div><div class="line">        <span class="keyword">synchronized</span> (<span class="keyword">this</span>) {</div><div class="line">            <span class="keywordflow">if</span> (mTexUpdate) {</div><div class="line">                mSTex.updateTexImage();</div><div class="line">                mTexUpdate = <span class="keyword">false</span>;</div><div class="line">            }</div><div class="line">        }</div><div class="line">        NativeGLRenderer.drawFrame();</div><div class="line">    }</div><div class="line"></div><div class="line">    @Override</div><div class="line">    <span class="keyword">public</span> <span class="keywordtype">void</span> onSurfaceChanged(GL10 gl, <span class="keywordtype">int</span> surfaceWidth, <span class="keywordtype">int</span> surfaceHeight) {</div><div class="line">        Log.i(LOGTAG, <span class="stringliteral">&quot;onSurfaceChanged(&quot;</span>+surfaceWidth+<span class="stringliteral">&quot;x&quot;</span>+surfaceHeight+<span class="stringliteral">&quot;)&quot;</span>);</div><div class="line">        NativeGLRenderer.changeSize(surfaceWidth, surfaceHeight);</div><div class="line">        setCameraPreviewSize(surfaceWidth, surfaceHeight);</div><div class="line">    }</div><div class="line"></div><div class="line">    @Override</div><div class="line">    <span class="keyword">public</span> <span class="keywordtype">void</span> onSurfaceCreated(GL10 gl, EGLConfig config) {</div><div class="line">        Log.i(LOGTAG, <span class="stringliteral">&quot;onSurfaceCreated&quot;</span>);</div><div class="line">        <a class="code" href="../../dc/d84/group__core__basic.html#ga1f6634802eeadfd7245bc75cf3e216c2">String</a> strGLVersion = GLES20.glGetString(GLES20.GL_VERSION);</div><div class="line">        <span class="keywordflow">if</span> (strGLVersion != null)</div><div class="line">            Log.i(LOGTAG, <span class="stringliteral">&quot;OpenGL ES version: &quot;</span> + strGLVersion);</div><div class="line"></div><div class="line">        <span class="keywordtype">int</span> hTex = NativeGLRenderer.initGL();</div><div class="line">        mSTex = <span class="keyword">new</span> SurfaceTexture(hTex);</div><div class="line">        mSTex.setOnFrameAvailableListener(<span class="keyword">this</span>);</div><div class="line">        openCamera();</div><div class="line">        mGLInit = <span class="keyword">true</span>;</div><div class="line">    }</div><div class="line">}</div></div><!-- fragment --><p>As you can see, inheritors for <code>Camera</code> and <code>Camera2</code> APIs should implement the following abstract methods: </p><div class="fragment"><div class="line"><span class="keyword">protected</span> <span class="keyword">abstract</span> <span class="keywordtype">void</span> openCamera();</div><div class="line"><span class="keyword">protected</span> <span class="keyword">abstract</span> <span class="keywordtype">void</span> closeCamera();</div><div class="line"><span class="keyword">protected</span> <span class="keyword">abstract</span> <span class="keywordtype">void</span> setCameraPreviewSize(<span class="keywordtype">int</span> width, <span class="keywordtype">int</span> height);</div></div><!-- fragment --><p>Let's leave the details of their implementation beyond of this tutorial, please refer the <a href="https://github.com/opencv/opencv/tree/master/samples/android/tutorial-4-opencl/">source code</a> to see them.</p>
<h2>Preview Frames modification </h2>
<p>The details OpenGL ES 2.0 initialization are also quite straightforward and noisy to be quoted here, but the important point here is that the OpeGL texture to be the target for camera preview should be of type <code>GL_TEXTURE_EXTERNAL_OES</code> (not <code>GL_TEXTURE_2D</code>), internally it keeps picture data in <em>YUV</em> format. That makes unable sharing it via CL-GL interop (<code>cl_khr_gl_sharing</code>) and accessing its pixel data via C/C++ code. To overcome this restriction we have to perform an OpenGL rendering from this texture to another regular <code>GL_TEXTURE_2D</code> one using <em>FrameBuffer Object</em> (aka FBO).</p>
<h3>C/C++ code</h3>
<p>After that we can read (<em>copy</em>) pixel data from C/C++ via <code>glReadPixels()</code> and write them back to texture after modification via <code>glTexSubImage2D()</code>.</p>
<h3>Direct OpenCL calls</h3>
<p>Also that <code>GL_TEXTURE_2D</code> texture can be shared with OpenCL without copying, but we have to create OpenCL context with special way for that:</p>
<div class="fragment"><div class="line"><span class="keywordtype">void</span> initCL()</div><div class="line">{</div><div class="line">    EGLDisplay mEglDisplay = eglGetCurrentDisplay();</div><div class="line">    <span class="keywordflow">if</span> (mEglDisplay == EGL_NO_DISPLAY)</div><div class="line">        LOGE(<span class="stringliteral">&quot;initCL: eglGetCurrentDisplay() returned &#39;EGL_NO_DISPLAY&#39;, error = %x&quot;</span>, eglGetError());</div><div class="line"></div><div class="line">    EGLContext mEglContext = eglGetCurrentContext();</div><div class="line">    <span class="keywordflow">if</span> (mEglContext == EGL_NO_CONTEXT)</div><div class="line">        LOGE(<span class="stringliteral">&quot;initCL: eglGetCurrentContext() returned &#39;EGL_NO_CONTEXT&#39;, error = %x&quot;</span>, eglGetError());</div><div class="line"></div><div class="line">    cl_context_properties props[] =</div><div class="line">    {   CL_GL_CONTEXT_KHR,   (cl_context_properties) mEglContext,</div><div class="line">        CL_EGL_DISPLAY_KHR,  (cl_context_properties) mEglDisplay,</div><div class="line">        CL_CONTEXT_PLATFORM, 0,</div><div class="line">        0 };</div><div class="line"></div><div class="line">    <span class="keywordflow">try</span></div><div class="line">    {</div><div class="line">        cl::Platform p = cl::Platform::getDefault();</div><div class="line">        std::string ext = p.getInfo&lt;CL_PLATFORM_EXTENSIONS&gt;();</div><div class="line">        <span class="keywordflow">if</span>(ext.find(<span class="stringliteral">&quot;cl_khr_gl_sharing&quot;</span>) == std::string::npos)</div><div class="line">            LOGE(<span class="stringliteral">&quot;Warning: CL-GL sharing isn&#39;t supported by PLATFORM&quot;</span>);</div><div class="line">        props[5] = (cl_context_properties) p();</div><div class="line"></div><div class="line">        theContext = cl::Context(CL_DEVICE_TYPE_GPU, props);</div><div class="line">        std::vector&lt;cl::Device&gt; devs = theContext.getInfo&lt;CL_CONTEXT_DEVICES&gt;();</div><div class="line">        LOGD(<span class="stringliteral">&quot;Context returned %d devices, taking the 1st one&quot;</span>, devs.size());</div><div class="line">        ext = devs[0].getInfo&lt;CL_DEVICE_EXTENSIONS&gt;();</div><div class="line">        <span class="keywordflow">if</span>(ext.find(<span class="stringliteral">&quot;cl_khr_gl_sharing&quot;</span>) == std::string::npos)</div><div class="line">            LOGE(<span class="stringliteral">&quot;Warning: CL-GL sharing isn&#39;t supported by DEVICE&quot;</span>);</div><div class="line"></div><div class="line">        theQueue = cl::CommandQueue(theContext, devs[0]);</div><div class="line"></div><div class="line">        <span class="comment">// ...</span></div><div class="line">    }</div><div class="line">    <span class="keywordflow">catch</span>(cl::Error&amp; e)</div><div class="line">    {</div><div class="line">        LOGE(<span class="stringliteral">&quot;cl::Error: %s (%d)&quot;</span>, e.what(), e.err());</div><div class="line">    }</div><div class="line">    <span class="keywordflow">catch</span>(std::exception&amp; e)</div><div class="line">    {</div><div class="line">        LOGE(<span class="stringliteral">&quot;std::exception: %s&quot;</span>, e.what());</div><div class="line">    }</div><div class="line">    <span class="keywordflow">catch</span>(...)</div><div class="line">    {</div><div class="line">        LOGE( <span class="stringliteral">&quot;OpenCL info: unknown error while initializing OpenCL stuff&quot;</span> );</div><div class="line">    }</div><div class="line">    LOGD(<span class="stringliteral">&quot;initCL completed&quot;</span>);</div><div class="line">}</div></div><!-- fragment --><dl class="section note"><dt>Note</dt><dd>To build this JNI code you need <b>OpenCL 1.2</b> headers from <a href="https://www.khronos.org/registry/cl/api/1.2/">Khronos web site</a> and the <b>libOpenCL.so</b> downloaded from the device you'll run the application.</dd></dl>
<p>Then the texture can be wrapped by a <code>cl::ImageGL</code> object and processed via OpenCL calls: </p><div class="fragment"><div class="line">cl::ImageGL imgIn (theContext, CL_MEM_READ_ONLY,  GL_TEXTURE_2D, 0, texIn);</div><div class="line">cl::ImageGL imgOut(theContext, CL_MEM_WRITE_ONLY, GL_TEXTURE_2D, 0, texOut);</div><div class="line"></div><div class="line">std::vector &lt; cl::Memory &gt; images;</div><div class="line">images.push_back(imgIn);</div><div class="line">images.push_back(imgOut);</div><div class="line">theQueue.enqueueAcquireGLObjects(&amp;images);</div><div class="line">theQueue.finish();</div><div class="line"></div><div class="line">cl::Kernel <a class="code" href="../../d4/d86/group__imgproc__filter.html#gad78703e4c8fe703d479c1860d76429e6">Laplacian</a> = ...</div><div class="line">Laplacian.setArg(0, imgIn);</div><div class="line">Laplacian.setArg(1, imgOut);</div><div class="line">theQueue.finish();</div><div class="line"></div><div class="line">theQueue.enqueueNDRangeKernel(Laplacian, cl::NullRange, cl::NDRange(w, h), cl::NullRange);</div><div class="line">theQueue.finish();</div><div class="line"></div><div class="line">theQueue.enqueueReleaseGLObjects(&amp;images);</div><div class="line">theQueue.finish();</div></div><!-- fragment --><h3>OpenCV T-API</h3>
<p>But instead of writing OpenCL code by yourselves you may want to use <b>OpenCV T-API</b> that calls OpenCL implicitly. All that you need is to pass the created OpenCL context to OpenCV (via <code><a class="el" href="../../dc/d83/group__core__opencl.html#gaf83af36be2097fd43e3eba5c8f3025b1" title="Attaches OpenCL context to OpenCV. ">cv::ocl::attachContext()</a></code>) and somehow wrap OpenGL texture with <code><a class="el" href="../../d7/d45/classcv_1_1UMat.html">cv::UMat</a></code>. Unfortunately <code>UMat</code> keeps OpenCL <em>buffer</em> internally, that can't be wrapped over either OpenGL <em>texture</em> or OpenCL <em>image</em> - so we have to copy image data here: </p><div class="fragment"><div class="line">cl::ImageGL imgIn (theContext, CL_MEM_READ_ONLY,  GL_TEXTURE_2D, 0, tex);</div><div class="line">std::vector &lt; cl::Memory &gt; images(1, imgIn);</div><div class="line">theQueue.enqueueAcquireGLObjects(&amp;images);</div><div class="line">theQueue.finish();</div><div class="line"></div><div class="line"><a class="code" href="../../d7/d45/classcv_1_1UMat.html">cv::UMat</a> uIn, uOut, uTmp;</div><div class="line"><a class="code" href="../../dc/d83/group__core__opencl.html#gada97cc08c31e416b957e23b480334b6c">cv::ocl::convertFromImage</a>(imgIn(), uIn);</div><div class="line">theQueue.enqueueReleaseGLObjects(&amp;images);</div><div class="line"></div><div class="line"><a class="code" href="../../d4/d86/group__imgproc__filter.html#gad78703e4c8fe703d479c1860d76429e6">cv::Laplacian</a>(uIn, uTmp, <a class="code" href="../../d1/d1b/group__core__hal__interface.html#ga32b18d904ee2b1731a9416a8eef67d06">CV_8U</a>);</div><div class="line"><a class="code" href="../../d2/d75/namespacecv.html">cv</a>:<a class="code" href="../../d2/de8/group__core__array.html#ga979d898a58d7f61c53003e162e7ad89f">multiply</a>(uTmp, 10, uOut);</div><div class="line"><a class="code" href="../../dc/d83/group__core__opencl.html#gabfca32247f267c662b3bff6587d14b40">cv::ocl::finish</a>();</div><div class="line"></div><div class="line">cl::ImageGL imgOut(theContext, CL_MEM_WRITE_ONLY, GL_TEXTURE_2D, 0, tex);</div><div class="line">images.clear();</div><div class="line">images.push_back(imgOut);</div><div class="line">theQueue.enqueueAcquireGLObjects(&amp;images);</div><div class="line">cl_mem clBuffer = (cl_mem)uOut.<a class="code" href="../../d7/d45/classcv_1_1UMat.html#afdbe2f205e6bb5475b252b598376188e">handle</a>(<a class="code" href="../../dc/d84/group__core__basic.html#gga6226c29c7b0e7bda7aff72e96ba8e1dfa39d5f615d02cac084ab1dd0cb4f7c221">cv::ACCESS_READ</a>);</div><div class="line">cl_command_queue q = (cl_command_queue)<a class="code" href="../../dd/dc0/classcv_1_1ocl_1_1Queue.html#a4f7a22c9c0f2b2c6ee6b47cc19519fef">cv::ocl::Queue::getDefault</a>().<a class="code" href="../../dd/dc0/classcv_1_1ocl_1_1Queue.html#a8d201148fb55a19e748b20b2f50a0610">ptr</a>();</div><div class="line"><span class="keywordtype">size_t</span> offset = 0;</div><div class="line"><span class="keywordtype">size_t</span> origin[3] = { 0, 0, 0 };</div><div class="line"><span class="keywordtype">size_t</span> region[3] = { w, h, 1 };</div><div class="line"><a class="code" href="../../db/de0/group__core__utils.html#gaf62bcd90f70e275191ab95136d85906b">CV_Assert</a>(clEnqueueCopyBufferToImage (q, clBuffer, imgOut(), offset, origin, region, 0, NULL, NULL) == CL_SUCCESS);</div><div class="line">theQueue.enqueueReleaseGLObjects(&amp;images);</div><div class="line"><a class="code" href="../../dc/d83/group__core__opencl.html#gabfca32247f267c662b3bff6587d14b40">cv::ocl::finish</a>();</div></div><!-- fragment --><ul>
<li><dl class="section note"><dt>Note</dt><dd>We have to make one more image data copy when placing back the modified image to the original OpenGL texture via OpenCL image wrapper.</dd></dl>
</li>
<li><dl class="section note"><dt>Note</dt><dd>By default the OpenCL support (T-API) is disabled in OpenCV builds for Android OS (so it's absent in official packages as of version 3.0), but it's possible to rebuild locally OpenCV for Android with OpenCL/T-API enabled: use <code>-DWITH_OPENCL=YES</code> option for CMake. <div class="fragment"><div class="line">cd opencv-build-android</div><div class="line">path/to/cmake.exe -GNinja -DCMAKE_MAKE_PROGRAM=&quot;path/to/ninja.exe&quot; -DCMAKE_TOOLCHAIN_FILE=path/to/opencv/platforms/android/android.toolchain.cmake -DANDROID_ABI=&quot;armeabi-v7a with NEON&quot; -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON path/to/opencv</div><div class="line">path/to/ninja.exe install/strip</div></div><!-- fragment --> To use your own modified <code>libopencv_java4.so</code> you have to keep inside your APK, not to use OpenCV Manager and load it manually via <code>System.loadLibrary("opencv_java4")</code>.</dd></dl>
<h2>Performance notes </h2>
</li>
</ul>
<p>To compare the performance we measured FPS of the same preview frames modification (<em>Laplacian</em>) done by C/C++ code (call to <code><a class="el" href="../../d4/d86/group__imgproc__filter.html#gad78703e4c8fe703d479c1860d76429e6" title="Calculates the Laplacian of an image. ">cv::Laplacian</a></code> with <code><a class="el" href="../../d3/d63/classcv_1_1Mat.html" title="n-dimensional dense array class ">cv::Mat</a></code>), by direct OpenCL calls (using OpenCL <em>images</em> for input and output), and by OpenCV <em>T-API</em> (call to <code><a class="el" href="../../d4/d86/group__imgproc__filter.html#gad78703e4c8fe703d479c1860d76429e6" title="Calculates the Laplacian of an image. ">cv::Laplacian</a></code> with <code><a class="el" href="../../d7/d45/classcv_1_1UMat.html">cv::UMat</a></code>) on <em>Sony Xperia Z3</em> with 720p camera resolution:</p><ul>
<li><b>C/C++ version</b> shows <b>3-4 fps</b></li>
<li><b>direct OpenCL calls</b> shows <b>25-27 fps</b></li>
<li><b>OpenCV T-API</b> shows <b>11-13 fps</b> (due to extra copying from <code>cl_image</code> to <code>cl_buffer</code> and back) </li>
</ul>
</div></div><!-- contents -->
<!-- HTML footer for doxygen 1.8.6-->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Apr 2 2021 11:36:36 for OpenCV by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="../../doxygen.png" alt="doxygen"/>
</a> 1.8.13
</small></address>
<script type="text/javascript">
//<![CDATA[
addTutorialsButtons();
//]]>
</script>
</body>
</html>
